System and method for applying a database to video multimedia

ABSTRACT

A system and method for applying a database to video multimedia is disclosed. Certain embodiments provide media content owners the capability to exploit video processing capabilities using rich, interactive and compelling visual content on a network. Mechanisms of associating video with commerce offerings are provided. Video server and search server technologies are integrated with ad serving personalization agents to make the final presentations of content and advertising. Algorithms utilized by the system use a variety of techniques for making the final presentation decisions of which ads, with which content, are served to which user.

RELATED APPLICATIONS

This application is a continuation application of U.S. application Ser.No. 10/872,191, filed Jun. 18, 2004, and issued as U.S. Pat. No.8,171,509, which is a divisional application of U.S. application Ser.No. 09/828,507, filed Apr. 6, 2001, which claims the benefit of U.S.Provisional Application No. 60/195,535, filed Apr. 7, 2000, each ofwhich are hereby incorporated by reference in their entirety.

This application is related to U.S. application Ser. No. 09/827,772,filed Apr. 6, 2001 and titled “SYSTEM AND METHOD FOR HOSTING OF VIDEOCONTENT OVER A NETWORK,” and issued as U.S. Pat. No. 7,222,163, U.S.application Ser. No. 09/828,618, filed Apr. 6, 2001 and titled“VIDEO-ENABLED COMMUNITY BUILDING,” and issued as U.S. Pat. No.7,962,948, and U.S. application Ser. No. 09/828,506, filed Apr. 6, 2001and titled “NETWORK VIDEO GUIDE AND SPIDERING,” and issued as U.S. Pat.No. 7,260,564, each of which are hereby incorporated by reference intheir entirety.

BACKGROUND

1. Field

The present invention generally relates to the field of applying adatabase to video multimedia. More particularly, the invention relatesto techniques for applying a database for accessing and processingdigital video on a network.

2. Description of the Related Technology

A number of techniques have evolved in recent years as the Internet hasgrown in size and sophistication, including:

-   -   The use of web servers and HTML delivery to web browsers.    -   The use of the application-server model for connecting database        information with web pages and interactive interfaces for end        users.    -   The use of dynamically generated HTML that pulls information        from a database to dynamically format HTML for delivery to the        end user.    -   The use of a template language to merge database output with        pre-formatted HTML presentations.    -   The use of ‘cookies’ to track individual user preferences as        they interact with the web pages and applications.    -   The use of e-commerce engines and financial transaction        processing technology (such as available from IBM, Qpass,        Oracle, etc.)    -   The use of agent technology to build and manage personalization        profiles (such as available from Autonomy, Semio, Cyber Dialog,        Net Perceptions, etc.)    -   The use of auto-categorization technologies to take a segment of        transcript or a document, and analyze it using natural language        processing techniques to identify category labels that apply to        the body of text. Example vendors of these technologies (which        also offer search technologies as well) include Webmind, Verity,        Autonomy, and Semio.

These and other related web technologies and techniques are incommonplace use and readily accessible on the Internet.

In addition to theses technologies, video indexing technology has alsoemerged, herein referred to as ‘video logging’. Video logging is aprocess that incorporates both automated indexing and manual annotationfacilities to create a rich, fine-grained (in a temporal sense) indexinto a body of video content. The index typically consists of acombination of visual and textual indices that permit time-basedsearching of video content. The index may incorporate spoken text,speaker identifications, facial identifications, on-screen text, andadditional annotations, keywords, and descriptions that may be appliedby a human user executing the video logging application. The VirageVideoLogger® is one example of this type of video logging technologythat is commercially available.

The delivery of streaming media on the Internet typically involves theencoding of video content into one or more streaming video formats andefficient delivery of that content for display to the end users. Commonstreaming formats presently in use include RealVideo, Microsoft WindowsMedia, QuickTime, and MPEG. The video logging technology may coordinatethe encoding of one or more of these formats while the video is beingindexed to ensure that the video index is time-synchronized with theencoded content. The final delivery of streaming media content fordisplay to an end user is typically accomplished with a wide variety ofvideo serving mechanisms and infrastructure. These mechanisms mayinclude basic video servers (such as those from Real, Microsoft, orApple), caching appliances (such as those from CacheFlow, NetworkAppliance, Inktomi, or Cisco), and content distribution networks (herein“CDN's”, such as those from Akamai, Digital Island, iBeam, or Adero).These types of video serving mechanisms ultimately deliver media contentfor display to the end user.

In an Internet/World Wide Web environment, companies frequently attemptto generate revenue though advertising and electronic commerce(hereinafter referred to as e-commerce) within their website. Whetherselling products, services, or advertising, they all have a primary needto engage visitors in a compelling presentation of their offering, orsomething associated with their offering that ultimately convinces thevisitor to make a purchase or follow an ad link, thus generating revenuefor the company.

Increased visitors, repeat visitors, and increased visitation time allcontribute to revenue streams derived from standard advertising models.In addition, these increased visitation properties also allow morenumerous and frequent opportunities for e-commerce (products andservices). E-commerce-based Websites exploiting video share the commongoal of using rich and interactive media content (such as video) to moreeffectively sell products and services. Compelling video content can beused to create web experiences that are more efficient and compelling interms of communicating value and relevance to the (potential) customer.Highly-targeted advertising and e-commerce is made possible byassociating demographic and product/service information with videocontent. Consumers are more likely to respond to targeted offerings thanrandom offerings, thus making the website more productive.

Therefore, what is needed in the technology is a system that effectivelyuses and manages video in a central role for commerce-oriented websitesso as to increase their success. What is desired are mechanisms ofassociating video with commerce offerings, which in turn, can be used tobuild the websites and e-commerce tools that many companies and websiteowners want.

SUMMARY OF CERTAIN INVENTIVE ASPECTS

The present system and method relate to techniques whereby varioustraditional mechanisms are combined in an innovative way with aninteractive video search and retrieval application environment. Videocontent is indexed and encoded using applications such as, for example,the VideoLogger available from Virage. The index provides a rich,fine-grained search mechanism to access the video in a non-linearfashion. This turns interactive video into a useful and attractivefeature on a website. The use of auto-categorization technology allowsthe system to automatically identify category designations of thecontent during the indexing phase, where the categories are useful inthe process of selecting relevant ads and commerce options to bepresented to the user. Thus, the index is structured to also providehigher level topic and category information.

A video search and retrieval application gives website visitors theability to search media content to find segments that are of interest.Utilizing these search and retrieval capabilities and a repository ofengaging content, various mechanisms can be added.

In one embodiment, there is a method of applying a database to videomultimedia, the method comprising indexing video content; storing theindexed video content in an index database, the indexed video contentcomprising metadata; encoding the video content concurrent with theindexing of the video content, wherein the index database does notcontain the encoded video content; and storing in the index database atleast one tag correlated with the video content on a time-code basis,wherein the tag is valid for a certain span of time within the video,and wherein the tag is configured to be associated with an advertisementor ecommerce opportunity, wherein the method is carried out in acomputing environment.

The method may additionally comprise making associations between thevideo content and at least one of ad banners, product offerings, andservice offerings so that such items are associated with the tags. Themethod may additionally comprise collecting a user profile describingthe content that is most of interest to the user. The method mayadditionally comprise learning the user profile by monitoring usagepatterns of the user. The user profile may be combined with the tags soas to make targeted associations between at least one of ads, products,services, and a person viewing the video content. The method mayadditionally comprise storing a plurality of indices that result fromthe indexing in the index database, wherein each stored index may beassociated with one of a plurality of different metadata types and atleast a portion of the stored indices are associated with different onesof the metadata types. The method may additionally comprisealgorithmically selecting a metadata element from a plurality ofmetadata elements in the user profile, wherein the algorithmic selectingutilizes one of cyclic, least-recently used, or random selection. Themethod may additionally comprise algorithmically selecting anadvertisement or ecommerce opportunity based on the selection of themetadata element. The algorithmic selecting of the advertisement orecommerce opportunity may utilize at least one of heuristics, fuzzylogic or hidden Markov models. The method may additionally comprisealgorithmically selecting an advertisement or ecommerce opportunitybased on selected metadata of the video content. The selectedadvertisement or ecommerce opportunity may be configured for displayconcurrently with viewing of video content that is played. The datacorresponding with a metadata type may have a time span that isdifferent than the data corresponding with another metadata type.

In another embodiment, there is a non-transitory computer readablemedium containing program instructions for applying a database to videomultimedia, wherein execution of the program instructions by a computingenvironment carries out a method, comprising indexing video content;storing a plurality of indices that result from the indexing in an indexdatabase, the indices comprising metadata; encoding the video contentconcurrent with the indexing of the video content, wherein the indexdatabase does not contain the encoded video content; storing in theindex database a plurality of tags correlated with the video content ona time-code basis via the index database; collecting, with apersonalization agent, a user profile describing the content that ismost of interest to the user; algorithmically selecting a singlemetadata type from a plurality of metadata types in the user profile;and algorithmically selecting an advertisement or ecommerce opportunityassociated with the selected single metadata type. The method embodiedby program instructions may additionally comprise combining the userprofile with the tags so as to make targeted associations between atleast one of ads, products, services, and the person viewing the videocontent. The method embodied by program instructions may additionallycomprise making associations between the video content and at least oneof ad banners, product offerings, and service offerings so that suchitems are associated with the tags.

In yet another embodiment, there is a system for applying a database tovideo multimedia, the system comprising a computing environmentconfigured to index video content; a computer database accessed by thecomputing environment, the computer database storing a plurality ofindices that result from indexing the video content, the indicescomprising metadata, wherein each stored index is associated with one ofa plurality of different metadata types and at least a portion of thestored indices are associated with different ones of the metadata types;the computing environment further configured to encode the video contentconcurrent with the indexing of the video content, wherein the databasedoes not contain the encoded video content; store in the database aplurality of tags correlated with the video content on a time-code basisvia the computer database; make associations between the video contentand at least one of ad banners, product offerings, and service offeringsso that such items are synchronized via the tags; and algorithmicallyselect an advertisement or ecommerce opportunity based on metadata ofthe video content. At least one of the tags may be valid for a certainspan of time within the video. The computing environment may be furtherconfigured to collect a user profile describing the content that is mostof interest to the user. The user profile may be combined with the tagsso as to make targeted associations between at least one of ads,products, services, and a person viewing the video content.

In yet another embodiment, there is a method of applying a database tovideo multimedia, the method comprising indexing video content togenerate an index; storing the index in an index database; encoding thevideo content concurrent with the indexing of the video content, whereinthe index database does not contain the encoded video content; andassociating a plurality of tags with the video content on a time-codebasis via the database, wherein at least one of the tags is valid for acertain span of time within the video.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages will be betterunderstood by referring to the following detailed description, whichshould be read in conjunction with the accompanying drawings. Thesedrawings and the associated description are provided to illustratecertain embodiments, and not to limit the scope of the invention.

FIG. 1 is a diagram of an example network configuration in which certainembodiments may operate.

FIG. 2 is a block diagram of an example system architecture inaccordance with certain embodiments.

FIG. 3 is a block diagram showing a high-level system view of theexample video application server embodiment shown in FIG. 2 and theserver's interaction with e-commerce subsystems.

FIG. 4 is a flowchart showing an example process of gathering andmanaging personalization profile information from the user to definetheir static personal profile such as performed on the examplearchitecture embodiment shown in FIG. 2.

FIG. 5 is a flowchart showing an example process of gathering andmanaging personalization profile information based on the user's viewinghabits to define their dynamic personal profile such as performed on theexample architecture embodiment shown in FIG. 2.

FIGS. 6 a, 6 b and 6 c are flowcharts showing an example delivery andresponse to a targeted e-commerce offering such as performed on theexample architecture embodiment shown in FIG. 2.

FIGS. 7 a, 7 b and 7 c are flowcharts showing example processes forusing content-based and personalization-based information to deliver atargeted advertisement such as performed on the architecture embodimentshown in FIG. 2.

DETAILED DESCRIPTION OF CERTAIN EMBODIMENTS

The following detailed description of certain embodiments presentsvarious descriptions of specific embodiments. However, the presentinvention can be embodied in a multitude of different ways as definedand covered by the claims. In this description, reference is made to thedrawings wherein like parts are designated with like numeralsthroughout.

Definitions

The following provides a number of useful possible definitions of termsused in describing certain embodiments.

A network may refer to a network or combination of networks spanning anygeographical area, such as a local area network, wide area network,regional network, national network, and/or global network. The Internetis an example of a current global computer network. Those terms mayrefer to hardwire networks, wireless networks, or a combination ofhardwire and wireless networks. Hardwire networks may include, forexample, fiber optic lines, cable lines, ISDN lines, copper lines, etc.Wireless networks may include, for example, cellular systems, personalcommunications service (PCS) systems, satellite communication systems,packet radio systems, and mobile broadband systems. A cellular systemmay use, for example, code division multiple access (CDMA), timedivision multiple access (TDMA), personal digital phone (PDC), GlobalSystem Mobile (GSM), or frequency division multiple access (FDMA), amongothers.

A website may refer to one or more interrelated web page files and otherfiles and programs on one or more web servers. The files and programsare accessible over a computer network, such as the Internet, by sendinga hypertext transfer protocol (HTTP) request specifying a uniformresource locator (URL) that identifies the location of one of said webpage files, wherein the files and programs are owned, managed orauthorized by a single business entity. Such files and programs caninclude, for example, hypertext markup language (HTML) files, commongateway interface (CGI) files, and Java applications. The web page filespreferably include a home page file that corresponds to a home page ofthe website. The home page can serve as a gateway or access point to theremaining files and programs contained within the website. In oneembodiment, all of the files and programs are located under, andaccessible within, the same network domain as the home page file.Alternatively, the files and programs can be located and accessiblethrough several different network domains.

A web page or electronic page may comprise that which is presented by astandard web browser in response to an HTTP request specifying the URLby which the web page file is identified. A web page can include, forexample, text, images, sound, video, and animation.

Content, media content and streaming media content may refer to thedelivery of electronic materials such as music, videos, software, books,multimedia presentations, images, and other electronic data, for exampleover a network to one or more users. Content data will typically be inthe form of computer files for video, audio, program, data and othermultimedia type content as well as actual physical copies of valuablecontent, for example CD-ROM, DVD, VCR, audio, TV or radio broadcastsignals, streaming audio and video over networks, or other forms ofconveying such information. The terms content, media content andstreaming media content may be used interchangeably.

A computer or computing device may be any processor controlled devicethat permits access to the Internet, including terminal devices, such aspersonal computers, workstations, servers, clients, mini-computers,main-frame computers, laptop computers, a network of individualcomputers, mobile computers, palm-top computers, hand-held computers,set top boxes for a television, other types of web-enabled televisions,interactive kiosks, personal digital assistants, interactive orweb-enabled wireless communications devices, mobile web browsers, or acombination thereof. The computers may further possess one or more inputdevices such as a keyboard, mouse, touch pad, joystick, pen-input-pad,and the like. The computers may also possess an output device, such as avisual display and an audio output. One or more of these computingdevices may form a computing environment.

These computers may be uni-processor or multi-processor machines.Additionally, these computers may include an addressable storage mediumor computer accessible medium, such as random access memory (RAM), anelectronically erasable programmable read-only memory (EEPROM),programmable read-only memory (PROM), erasable programmable read-onlymemory (EPROM), hard disks, floppy disks, laser disk players, digitalvideo devices, compact disks, video tapes, audio tapes, magneticrecording tracks, electronic networks, and other techniques to transmitor store electronic content such as, by way of example, programs anddata. In one embodiment, the computers are equipped with a networkcommunication device such as a network interface card, a modem, or othernetwork connection device suitable for connecting to the communicationnetwork. Furthermore, the computers execute an appropriate operatingsystem such as Linux, Unix, any of the versions of Microsoft Windows,Apple MacOS, IBM OS/2 or other operating system. The appropriateoperating system may include a communications protocol implementationthat handles all incoming and outgoing message traffic passed over theInternet. In other embodiments, while the operating system may differdepending on the type of computer, the operating system will continue toprovide the appropriate communications protocols to establishcommunication links with the Internet.

The computers may contain program logic, or other substrateconfiguration representing data and instructions, which cause thecomputer to operate in a specific and predefined manner, as describedherein. In one embodiment, the program logic may be implemented as oneor more object frameworks or modules. These modules may be configured toreside on the addressable storage medium and configured to execute onone or more processors. The modules include, but are not limited to,software or hardware components that perform certain tasks. Thus, amodule may include, by way of example, components, such as, softwarecomponents, object-oriented software components, class components andtask components, processes, functions, attributes, procedures,subroutines, segments of program code, drivers, firmware, microcode,circuitry, data, databases, data structures, tables, arrays, andvariables.

The various components of the system may communicate with each other andother components comprising the respective computers through mechanismssuch as, by way of example, interprocess communication, remote procedurecall, distributed object interfaces, and other various programinterfaces. Furthermore, the functionality provided for in thecomponents, modules, and databases may be combined into fewercomponents, modules, or databases or further separated into additionalcomponents, modules, or databases. Additionally, the components,modules, and databases may be implemented to execute on one or morecomputers. In another embodiment, some of the components, modules, anddatabases may be implemented to execute on one or more computersexternal to the website. In this instance, the website includes programlogic, which enables the website to communicate with the externallyimplemented components, modules, and databases to perform the functionsas disclosed herein.

Overview of Auto-categorization

Auto-categorization of content, specifically applying a category to agiven time segment of the video, is particularly useful in certainembodiments. Video logging applications (such as the Virage VideoLogger)can typically accept novel signal and linguistic analysis algorithms tofurther refine and extend the metadata index generated during thelogging phase. Several vendors offer categorization algorithms andtechnology, such as Webmind, Verity, Autonomy, and Semio. Extensibilityand how it is used to integrate additional technology, such ascategorization, is described in Applicant's copending U.S. Patentapplication Ser. No. 09/134,497, entitled “Video Cataloger System WithSynchronized Encoders”, which is hereby incorporated by reference.Categorization technology from any of these vendors can thus beintegrated into the logging phase. Each of these technologies accepts aninput stream of text and responds with a category designation. Thecategories are used in the process of selecting relevant ads andcommerce options to be presented to the user. Most of these offeringsrequire a training phase whereby a known body of content andcorresponding categories are provided to the categorization engine, anda linguistic model is constructed. Thereafter, as new content issubmitted to the engine, it can reliably generate category designations.These systems are effective across multiple languages, and arerelatively new and rapidly maturing. Auto-categorization of content isutilized because it offers the ability to scale the content processingup to large volumes within an automatic process. Manual solutions arealso available (e.g., human editors making judgment calls on thecontent) but are much less scalable in a business sense.

Overview of E-commerce Mechanisms

A video search and retrieval (e.g., browse) application gives websitevisitors the ability to search media content to find segments that areof interest. Utilizing these search and retrieval capabilities and arepository of engaging content, various e-commerce mechanisms can beadded on. Ad banners, product offerings, and service offerings can eachbe triggered to appear in a synchronized fashion with video contentbeing viewed by the end-user. For example, a product demonstration videocan be viewed with associated links and mechanisms to purchase theproduct. A sports video can have sneaker ads automatically interspersed.These associations are made possible by associating keyword ‘tags’ withvideo content on a time-code basis. The tag is ‘valid’ for a certainspan of time within the video. A metadata model, time spans, time stampsand other related concepts are further described in Applicant'scopending U.S. patent application Ser. No. 09/134,497, entitled “VideoCataloger System With Synchronized Encoders”, especially in conjunctionwith FIGS. 6, 7, 8 and 9 of the application.

A related mechanism for targeting advertising and e-commerce is alsodisclosed. Given a repository of indexed video as described above, it isalso possible to create ‘personalization agents’ to exploit userinformation, in addition to content-based information, in the targetingprocess. A personalization agent gathers a specification (a ‘profile’)from the user describing the content, products, and servers that aremost of interest to the user. Additionally, a personalization agent hasthe ability to ‘learn’ the personalization profile by monitoring theusage patterns of the user. The personalization profile, combined withcontent-based tagging, can be used to make highly-targeted associationsbetween ads, products, services, and the person viewing the content.

Video server and search server technologies are integrated with adserving personalization agents to make the final presentations ofcontent, advertising, and commerce. The algorithms for making the finalpresentation decisions (which ads with which content served to whichuser) may be made using combinations of any of the following: look-uptables, keyword intersections, heuristics, fuzzy-logic, Hidden MarkovModels (HMM's), and so forth.

DESCRIPTION OF THE FIGURES

FIG. 1 is a diagram of an example network configuration 100 in whichcertain embodiments may operate. However, various other types ofelectronic devices communicating in a networked environment may also beused. An end user 102 communicates with a computing environment, whichmay include multiple server computers 108 or a single server computer110 in a client/server relationship on a network communication medium116. In a typical client/server environment, each of the servercomputers 108, 110 may include a server program that communicates with auser device 115, which may be a personal computer (PC), a hand-heldelectronic device, a mobile or cellular phone, a TV set or any number ofother electronic devices.

The server computers 108, 110, and the user device 115 may each have anyconventional general purpose single- or multi-chip microprocessor, forexample a Pentium processor, a Pentium Pro processor, a MIPS processor,a Power PC processor, an ALPHA processor, or other processor. Inaddition, the microprocessor may be any conventional special purposemicroprocessor such as a digital signal processor or a graphicsprocessor. Additionally, the server computers 108, 110 and the userdevice 115 may be desktop, server, portable, hand-held, set-top, orother desired type of computing device. Furthermore, the servercomputers 108, 110 and the user device 115 each may be used inconnection with various operating systems, including, for example, UNIX,LINUX, Disk Operating System (DOS), VxWorks, PalmOS, OS/2, any versionof Microsoft Windows, or other operating system.

The server computers 108, 110 and the user device 115 may each include anetwork terminal equipped with a video display, keyboard and pointingdevice. In one embodiment of the network configuration 100, the userdevice 115 includes a network browser 120 used to access the servercomputers 108,110. The network browser 120 may be, for example,Microsoft Internet Explorer or Netscape Navigator. The user 102 at theuser device 115 may utilize the browser 120 to remotely access theserver program using a keyboard and/or pointing device and a visualdisplay, such as a monitor 118. Although FIG. 1 shows only one userdevice 115, the network configuration 100 may include any number ofclient devices.

The network 116 may be any type of electronic transmission medium, forexample, including but not limited to the following networks: a virtualprivate network, a public Internet, a private Internet, a secureInternet, a private network, a public network, a value-added network, anintranet, or a wireless gateway. The term “virtual private network”refers to a secure and encrypted communications link between nodes onthe Internet, a Wide Area Network (WAN), Intranet, or any other networktransmission means.

In addition, the connectivity to the network 116 may be via, forexample, a modem, Ethernet (IEEE 802.3), Token Ring (IEEE 802.5), FiberDistributed Datalink Interface (FDDI), Asynchronous Transfer Mode (ATM),Wireless Application Protocol (WAP), or other form of networkconnectivity. The user device 115 may connect to the network 116 by useof a modem or by use of a network interface card that resides in theuser device 115. The server computers 108 may be connected via a widearea network 106 to a network gateway 104, which provides access to thewide area network 106 via a high-speed, dedicated data circuit.

As would be understood by one skilled in the technology, devices otherthan the hardware configurations described above may be used tocommunicate with the server computers 108, 110. If the server computers108, 110 are equipped with voice recognition or Dual ToneMulti-Frequency (DTMF) hardware, the user 102 may communicate with theserver computers by use of a telephone 124. The telephone may optionallybe equipped with a browser 120 and display screen. Other examples ofconnection devices for communicating with the server computers 108, 110include a portable personal computer (PC) 126 or a personal digitalassistant (PDA) device with a modem or wireless connection interface, acable interface device 128 connected to a visual display 130, or asatellite dish 132 connected to a satellite receiver 134 and atelevision 136. Still other methods of allowing communication betweenthe user 102 and the server computers 108, 110 are additionallycontemplated by this application.

Additionally, the server computers 108, 110 and the user device 115 maybe located in different rooms, buildings or complexes. Moreover, theserver computers 108, 110 and the user device 115 could be located indifferent geographical locations, for example in different cities,states or countries. This geographic flexibility which networkedcommunications allows is additionally within the contemplation of thisapplication.

FIG. 2 is a block diagram of an example system architecture 200 inaccordance with certain embodiments. In one embodiment, the systemarchitecture 200 includes a commerce website facility 210, which furtherincludes a video encoding module 214 and a video logging module 216,both of which receives media content 212, in one embodiment. Althoughthe term facility is used, the components do not necessarily need to beat a common location. The commerce website facility 210 further includesa video editorial module 218, which communicates with the video loggingmodule 216. The commerce website facility 210 further includes a videoapplication server 220, which communicates with the video editorialmodule 218. The commerce website facility 210 further includes a webserver 222, which communicates with the video application server 220.The commerce website facility 210 further includes a video index 224,which is produced by the video logging module 216 and the videoeditorial module 218, and is maintained by the video application server220. The commerce website facility 210 further includes a serveradministration (“Admin”) module 228, which communicates with the webserver module 222. The commerce website facility 210 further includes acommerce module 250 and a personalization module 260, both of whichcommunicate with the video application server 220 and the web server222. The commerce module 250 and the personalization module 260 aredescribed in greater detail below with regards to certain embodiments ofFIG. 2, and additionally in reference to FIGS. 4 through 7.

In one embodiment, the system architecture 200 further includes thenetwork 116 shown in FIG. 1, which may be the Internet. Web pages 232and search forms 234 are accessible via the Internet 116. Each web page232 may depict a plurality of pages rendered by various web servers. Thesearch form 234 is also accessible by the commerce website facility webserver 222. Additionally accessible via the Internet 116 is results data238, which is produced by the web server 222. Also accessible via theInternet 116 is a video player 236, which communicates with the webserver 222. The system architecture 200 further includes a contentdistribution network 240, which transfers encoded video to the videoplayer 236. The content distribution network 240 further receivesuploaded digital video files from the video encoding module 214. Thecontent distribution network 240 may be part of a wide variety of videoserving mechanisms and infrastructures that serve to deliver encodedmedia content 242 for display to the end user 102 shown in FIG. 1. Thecontent distribution network 240 may include a content owner running asimple video server at the content owner facility 220, a complex edgecaching content distribution mechanism, or other mechanisms to transmitvideo and other media content for display to end users 102.

The following paragraphs provide a description of the operation of oneembodiment of the system architecture 200 shown in FIG. 2. A commercewebsite may be hosted internally on the commerce web server 222 as shownin FIG. 2, or alternatively outsourced to a web-hosting serviceprovider, which delivers commerce features as described herein to endusers 102. The operation of the video encoding module 214, video loggingmodule 216, video editorial module 218, video application server module220, video index 224, administration module 228, and web server 222 aredescribed with respect to embodiments disclosed in the relatedapplication titled “Interactive Video Application Hosting” (U.S.application Ser. No. 09/827,772), which was incorporated by referenceabove. To the extent that these modules may operate differently incertain embodiments than in the related application, any suchdifferences will be described herein.

In one embodiment, the video application server module 220 manages thevideo index containing metadata and annotations produced by the videologging module 216. The application server 220 receives video andmetadata after the video logging 216 and video editorial 218 modules,and transfers video search form 234 queries and results 238 data to theweb server 222 for display to an end user 102 (FIG. 1) in a web browser120 at the user device 115 via the Internet 116. In one embodiment, thecommunication of the search form 234 queries and results 238 data to theweb server 222 includes an exchange of extensible markup language (XML)data, although one skilled in the technology will understand that otherdata exchange formats may also be utilized. Final HTML rendering ofsearch foams 234, results 238 presentation, and video player 236playback windows may be accomplished via templates, whereby suchtemplates dictate the graphical look-and-feel of the final mediapresentation. Actual metadata results, communicated via XML or otherdata exchange formats, may be rendered into the template by substitutingspecial keywords with results from the video application server 220 toform an HTML-compliant presentation. Additional communications may beprovided with the administration module 228 for server administration,metadata editing, and batch processing. Batch processing may beaccomplished for insertion processes, deletion or ‘aging’ processes,metadata editing, or for automated performance of other tasks as well.The administration module 228 further allows system administrators tomanage the video application server 220, including, for example, indexmanagement, asset management, editing, and startup and shutdown control.

In one embodiment, regardless of its original form, the content 212 isprocessed by the video logging module 216 to extract index data, forexample keyframes, closed-caption text, speaker identifications, facialidentifications, or other index data. The content 212 may additionallyundergo processing by the video editorial module 218, whereby humans mayelect to add labels to the index of the content 212 by providingadditional annotations, descriptions, keywords, or any other markinginformation such as commerce tags. The index and annotation informationis transferred to the video application server 220, which hostspublishing, search, retrieval, browse, or other related video services.The video application server 220 may maintain the metadata in the videoindex 224. The video application server 220 provides the above-describedvideo services to the web server 222 for incorporation into the webpages 232 via the template mechanism described above.

In another embodiment, the video application server 220 includes theserver itself that processes XML-based queries and data managementactivities, performs searches against the video index, and returns videoreferences and metadata via XML or other data exchange formats. Othermodules of the video application server 220 include the search or browserendering interface which processes HTML requests into XML, andadditionally processes XML responses back into HTML for delivery by theweb server 222 using templates to format and render the XML data intoHTML.

In one embodiment, the video application server's 220 XML-based openarchitecture allows for simple integration of additional features andfunctions, such as, for example, an e-commerce engine as shown in FIG.3. Such functions may be implemented in various commonly usedprogramming languages, for example Perl, C, C++, Java, or otherprogramming languages, and may utilize publically or commerciallyavailable packages for parsing and formatting XML or other data exchangeformats.

FIG. 3 is a block diagram showing a high-level view 300 of the videoapplication server (VAS) and it's interaction with e-commerce, targetedadvertising, and personalization subsystems. A video application serverarchitecture includes the server that processes XML-based queries anddata management activities, performs searches against the video index,and returns video references and metadata via XML. One such architectureis described in U.S. application Ser. No. 09/827,772, filed Apr. 6, 2001and titled “Interactive Video Application Hosting” and which wasincorporated by reference above. Other modules of the application serverinclude a Search/Browse rendering interface which processes HTMLrequests into XML, and also processes XML responses back into HTML fordelivery by the Web server using templates to format and render the XMLdata into HTML; and, the Administration module that allows systemadministrators to manage the application server (index management, assetmanagement, editing, start-up/shut-down, etc.).

The video application server's open, XML-based, architecture readilyallows the integration of additional features and functions, fromsyndication engines, commerce building mechanisms, to the e-commerce,targeted advertising systems, and personalization modules contemplatedhere. Any such modules can be implemented in any of several commonlyused languages (Perl, C, C++, Java, etc.), and can utilize publicallyand commercially available packages of subroutines for parsing andformatting XML.

The Personalization Server in FIG. 3 interacts with the PersonalizationInterface through any of a number of communication mechanisms, includingHTML, XML, and proprietary protocols specific to the PersonalizationServer employed. The main task of the Personalization Interface is tomediate between the protocol and semantic vocabulary of the chosenPersonalization Server and the Video Application Server's XML interface.The VAS serves as a persistent store of state information aboutindividual users to maintain profiles on behalf of the PersonalizationServer. Personalization features are rendered in HTML for the end user,which allows the user to select categories, topics, and preferences tohelp define their individual profile. Some Personalization Servers willalso allow for monitoring of individual's activity and behavior to moreaccurately characterize the preferences of the end user. This isreferred to as ‘learning behavior’ and allows the personal profile togrow and change over time. The system can accommodate a range ofcapabilities within the Personalization Server, and can supply contextand monitoring information about the user in question. Typically,user-specific profiles and behavior information can be stored within a“cookie” on the user's own computer, set-top box, etc., thus insuringprivacy. The value of personalization technology relevant to the systemis in its ability to direct and select the presentation of e-commerceand advertising opportunities for the end user. The PersonalizationInterface module is the connective mechanism between the detailedinformation about the user, the detailed information about the content(based on the automatic indexing), and the range of available commerceand advertising opportunities that could be presented to the user at anygiven point in time, based on the content being viewed and the user inquestion.

The e-commerce engine embodiment shown in FIG. 3 represents any of anumber of commercially available engines for processing e-commercetransactions by interfacing with standard transaction infrastructure,represented by the Transaction System module. The Transaction System inreality represents the diverse processing subsystems typified byofferings from SAP and others present in many commercial enterprises.The Transaction System interfaces with databases, order processingsubsystems, shipping, inventory, billing, and customer services systems.In the system, the e-commerce interface is responsible for mediatingbetween the various information sources (personalization and videocontent via the Video Application Server) that determine whiche-commerce opportunity should be presented to the end user at any givenpoint in time, based on the video content and the preferences of theuser. The e-commerce opportunity is presented to the user in an HTMLframework, and should the user select a commercial transaction, controlis passed to the e-commerce engine from the e-commerce interface.

The ad server mechanism embodiment shown in FIG. 3 represents any of anumber of commercial ad server vendors, most of which offer facilitiesfor requesting a topic-specific ad in response to a request thatcontains category information. The primary task of the ad interface isto mediate between the content-specific and personal profile-specificinformation (via XML) and the protocol of the ad server. The result of arequest is a targeted ad (in the form of a banner, video clip, etc.)that is presented to the user via HTML in context with the video clipbeing served by the video application server.

FIG. 4 is a flowchart showing a process 400 of gathering and managingpersonalization profile information from the user to define their staticpersonal profile. The exact mechanism and profile information gatheredin this process 400 is dependent on the personalization server employedin the system; the process 400 depicted in FIG. 4 is merely offered asan example of the types of profile information that can be gathered andthe manner in which a system might interact with a user to gather suchinformation. The process 400 begins when the user elects to define ormodify their personal profile. Typically, the user may selectcategories, topics within those categories, and arbitrary keywords todefine their static profile. This profile information is activelydefined by the user, and is stored on their behalf, being relativelystatic in the sense that it does not dynamically update based on theirviewing habits. Categories are typically selected from a pre-definedlist of available categories, and might include things like ‘politics’,‘sports’, ‘science’, etc. Topics are a further refinement within acategory, and might include things like ‘presidential elections’,‘hockey highlights’, or ‘the moons of Jupiter’. Topic selection firstbegins by selecting a category within which specific topics are selectedfrom pre-defined lists. Keywords, unlike categories and topics, arespecified with a free-form entry, and are not pre-defined. Keywords aretypically any word or set of words that the user deems of interest tothem that might appear in the transcript of the video. Examples ofkeywords include proper nouns (persons, places, locations, ororganizations) and other nouns that carry information important to theuser. A given personalization server might employ any or all of thesemethods of defining a personal profile. Additionally, somepersonalization servers may also allow the specification of a weightingmechanism to identify the importance of each selection. For example,‘science’ may be more important to the user than ‘politics’, and theuser will be offered the ability to indicate this distinction through animportance rating (High, Medium, Low) or a numerical weighting value. Inone embodiment, the profile information is stored on the user's behalfusing a standard ‘cookie’ mechanism to maintain the profile on theuser's local computer, thus insuring privacy. The VAS can then lateraccess this information when the user's profile is required for commerceor advertising purposes.

FIG. 5 is a flowchart showing a process 500 of gathering and managingpersonalization profile information based on the user's viewing habitsto define their dynamic personal profile. The dynamic profile isconstantly updated based on the video content that the user views. Theprocess 500 is invoked whenever the user proactively searches forcontent and chooses to view it. At this point, the video metadatapreviously extracted during the indexing process is consulted to extractcategory, topic, and keyword information that can contribute to theuser's dynamic personal profile. This information is readily availableas part of the video index, and can be easily gathered and added to thedynamic profile of the user. The dynamic profile is stored and accessedusing the standard ‘cookie’ mechanism previously described for thestatic profile process 400 described in conjunction with FIG. 4.

FIGS. 6 a, 6 b and 6 c are flowcharts showing the delivery and responseto a targeted e-commerce offering. FIG. 6 a illustrates a process 600 ofmaking a targeted e-commerce opportunity available based on the subjectinformation of the video being viewed at that moment by the user. FIG. 6b illustrates a similar process 620 based on using the personal profileinformation of the user. FIG. 6 c illustrates a combined process 650 ofusing both the video content and the personal profile information tomake an e-commerce opportunity available. In each case, an opportunityto make a purchase of a product or service is offered to the user inconjunction with viewing a video. This is similar to advertising intraditional broadcast video, but with two important differences. Thefirst is that the commerce opportunity is offered concurrently with theviewing of the video. The second is that it is more than anadvertisement; if the user selects the opportunity (either interruptingtheir viewing experience, or after their viewing experience iscomplete), the user can actually complete a purchase on the spot.

The process 600 shown in FIG. 6 a begins with the user viewing aselected video clip. The video index is then consulted to extract thecorresponding category information for that clip. The category (forexample, ‘sports’) is submitted to the e-commerce server to request acommerce opportunity corresponding to the category (for example, ahockey highlights video for purchase). The commerce server returns thepurchase opportunity, typically in the form of a graphic description ofthe highlights video available for purchase. If the user clicks on theopportunity, the purchase transaction is forwarded to the commerceserver for fulfillment. At this point, detailed purchase information isgathered by the commerce server (such as DVD or video tape, billing andshipping information, etc.), and the commerce transaction is completed.

The process 620 shown in FIG. 6 b is similar to the process 600 in FIG.6 a, except that the category information is extracted from thepersonalization profile(s) of the user. In this case, more than onecategory selection is usually present. Therefore, the process 620includes a step to make a single category selection from the pluralityof categories present in the personal profile. The selection mechanismcan be any of a number of algorithms, including random selection (usinga random number generator), cyclic (or ‘round-robin’ selection), orleast-recently-used. The selected category is then submitted to thecommerce server, and the transaction continues as for FIG. 6 a.

FIG. 6 c depicts the combined process 650 that uses the video categoryinformation in conjunction with the personal profile information. Inthis case, the system attempts to make a match between the videocategory and any of the categories present in the personal profile. If amatch is found, the matching category is submitted to the commerceserver as before. If no match is found, the selection mechanism (random,cyclic, etc.) is used to select a category, and the process 650 proceedsas before.

FIGS. 7 a, 7 b and 7 c are flowcharts showing processes usingcontent-based and personalization-based information to deliver atargeted advertisement. FIG. 7 a illustrates a process 700 of making atargeted advertising available based on the subject information of thevideo being viewed at that moment by the user. FIG. 7 b illustrates asimilar process 720 based on using the personal profile information ofthe user. FIG. 7 c illustrates a combined process 750 of using both thevideo content and the personal profile information to make anadvertisement available. In each case, an advertisement is offered tothe user in conjunction with viewing a video. Typical advertisements canbe clicked upon by the user to find out more information, be transportedto another website, and potentially make a purchase there.

The process 700 shown in FIG. 7 a begins with the user viewing aselected video clip. The video index is then consulted to extract thecorresponding category information for that clip. The category (forexample, ‘science’) is submitted to the advertising server to request anadvertisement corresponding to the category (for example, ‘come learnabout space at Space.com). The ad server returns the advertisement,typically in the form of a clickable banner ad or video clip. If theuser clicks on the ad, their browser typically connects to anotherwebsite pertaining to the advertisement.

The process 720 shown in FIG. 7 b is similar to the one in FIG. 7 a,except that the category information is extracted from thepersonalization profile(s) of the user. In this case, more than onecategory selection is usually present. Therefore, the process 720includes a step to make a single category selection from the pluralityof categories present in the personal profile. The selection mechanismcan be any of a number of algorithms, including random selection (usinga random number generator), cyclic (or ‘round-robin’ selection), orleast-recently-used. The selected category is then submitted to theadvertising server, and the process 720 continues as described for FIG.7 a.

FIG. 7 c depicts a combined process 750 that uses the video categoryinformation in conjunction with the personal profile information. Inthis case, the system attempts to make a match between the videocategory and any of the categories present in the personal profile. If amatch is found, the matching category is submitted to the advertisingserver as before. If no match is found, the selection mechanism (random,cyclic, etc.) is used to select a category, and the process 750 proceedsas before.

Embodiments of the system and method may use:

-   -   video indexing tools to automatically extract textual metadata        used in search processes and to generate categories        automatically for commerce associations.    -   video indexing tools to carefully place commerce tags as        time-stamped elements to be associated with the video content        during playback.    -   personalization agents to gather and generate user profiles and        demographic information to be consulted by the video server        technology.    -   video and search serving technology to exploit the commerce tags        in the video content and the personalization profile of the user        watching the video to make decisions about which ads, products,        and/or services should be presented to the user.    -   the presence of commerce tags in the video stream combined with        personalization profiles to allow interaction with viewers in a        highly targeted manner so as to achieve true, 1-to-1 marketing        and sales on large populations.

As described herein, certain embodiments fill the longstanding need inthe technology of a system that provides commerce oriented websites thecapability to achieve their e-commerce goals by exploiting videoprocessing capabilities using rich and interactive media content. Whilethe above detailed description has shown, described, and pointed outnovel features of the invention as applied to various embodiments, itwill be understood that various omissions, substitutions, and changes inthe form and details of the device or process illustrated may be made bythose skilled in the art without departing from the intent of theinvention. The scope of the invention is indicated by the appendedclaims rather than by the foregoing description. All changes that comewithin the meaning and range of equivalency of the claims are to beembraced within their scope.

1-20. (canceled)
 21. A method of delivering targeted advertisement, saidmethod comprising: determining video content being viewed by a user;determining a category of at least one of the video content and theuser; accessing an advertisement corresponding to the determinedcategory; and displaying the accessed advertisement to the user.
 22. Themethod according to claim 21, wherein determining the category furthercomprises consulting a video index of the video content to extract thecategory corresponding to the video content being viewed by the user.23. The method according to claim 22, wherein consulting the video indexof the video content further comprises consulting a database on whichthe video index of the video content is stored.
 24. The method accordingto claim 23, wherein the database also stores a tag correlated to thevideo content on a time-code basis, wherein the tag is valid for acertain span of time within the video content, wherein the tag isassociated with a particular advertisement, wherein consulting thedatabase further comprises consulting the database to extract the tag,and wherein accessing the advertisement further comprises accessing theparticular advertisement associated with the extracted tag.
 25. Themethod according to claim 22, further comprising: accessing a userprofile category of the user, wherein the user profile categorycorresponds to a particular category; determining whether there is amatching category between the extracted category and the particularcategory of the user profile category; and wherein determining thecategory further comprises determining the category to be the matchingcategory in response to a determination that there is a matchingcategory.
 26. The method according to claim 25, wherein determining thecategory further comprises selecting the category from a plurality ofavailable categories through implementation of a selection technique inresponse to a determination that there is no matching category.
 27. Themethod according to claim 21, further comprising: accessing a userprofile of the user; and wherein determining the category furthercomprises determining the category based upon the user profile of theuser.
 28. The method according to claim 21, wherein accessing theadvertisement corresponding to the determined category furthercomprises: submitting the determined category to one of an advertisingserver and an e-commerce server, wherein the advertising server or thee-commerce server is to select the advertisement from the determinedcategory; and receiving the selected advertisement from the advertisingserver or the e-commerce server.
 29. The method according to claim 21,wherein displaying the accessed advertisement to the user furthercomprises displaying the accessed advertisement concurrently with thevideo content being viewed by the user.
 30. The method according toclaim 21, further comprising: receiving a selection of the displayedadvertisement; and directing the user to a server associated with thedisplayed advertisement.
 31. An apparatus for delivering targetedadvertising, said apparatus comprising: a memory on which is storedmachine readable instructions to: determine video content being viewedby a user; determine a category corresponding to at least one of thevideo content and the user; access an advertisement corresponding to thedetermined category; and display the accessed advertisement to the user;and a processor to execute the machine readable instructions.
 32. Theapparatus according to claim 31, wherein, to determine the categorycorresponding to at least one of the video content and the user, themachine readable instructions are further to: determine a tag correlatedto the video content on a time-code basis, wherein the tag is valid fora certain span of time within the video content and is associated with aparticular advertisement; and wherein, to access the advertisement, themachine readable instructions are further to access the advertisementthat is associated with the determined tag.
 33. The apparatus accordingto claim 31, wherein the machine readable instructions are further to:access a user profile category of the user, wherein the user profilecategory corresponds to a particular category; determine whether thereis a matching category between the determined category and theparticular category of the user profile category; and wherein, todetermine the category, the machine readable instructions are further todetermine the category to be the matching category in response to adetermination that there is a matching category.
 34. The apparatusaccording to claim 33, wherein, to determine the category, the machinereadable instructions are further to: select the category from aplurality of available categories through implementation of a selectiontechnique in response to a determination that there is no matchingcategory.
 35. The apparatus according to claim 31, wherein the machinereadable instructions are further to: access a user profile of the user;and wherein, to determine the category, the machine readableinstructions are further to determine the category based upon the userprofile of the user.
 36. The apparatus according to claim 31, whereinthe machine readable instructions are further to display the accessedadvertisement concurrently with the video content being viewed by theuser.
 37. The apparatus according to claim 31, wherein the machinereadable instructions are further to: receive a selection of thedisplayed advertisement; and direct the user to a server associated withthe displayed advertisement.
 38. The apparatus according to claim 31,wherein the advertisement comprises at least one of an ad banner, aproduct offering, and a service offering.
 39. A non-transitory computerreadable storage medium on which is stored machine readable instructionsthat when executed by a processor are to: determine video content beingviewed by a user; consult a database containing information pertainingto at least one of the video content and the user to determine acategory corresponding to at least one of the video content and theuser; access an advertisement corresponding to the determined category;and display the accessed advertisement to the user.
 40. Thenon-transitory computer readable storage medium according to claim 39,wherein the database also has stored thereon a tag correlated to thevideo content on a time-code basis, wherein the tag is valid for acertain span of time within the video content and is associated with aparticular advertisement, and wherein, to access the advertisement, themachine readable instructions are further to: access the advertisementthat is associated with the determined tag.